System for surveillance by integrating radar with a panoramic staring sensor

ABSTRACT

Described is system for surveillance that integrates radar with a panoramic staring sensor. The system captures image frames of a field-of-view of a scene using a multi-camera panoramic staring sensor. The field-of-view is scanned with a radar sensor to detect an object of interest. A radar detection is received when the radar sensor detects the object of interest. A radar message indicating the presence of the object of interest is generated. Each image frame is marked with a timestamp. The image frames are stored in a frame storage database. The set of radar-based coordinates from the radar message is converted into a set of multi-camera panoramic sensor coordinates. A video clip comprising a sequence of image frames corresponding in time to the radar message is created. Finally, the video clip is displayed, showing the object of interest.

CROSS-REFERENCE TO RELATED APPLICATIONS

This is a Continuation-in-Part application of U.S. Non-Provisionalapplication Ser. No. 13/743,742, filed in the United States on Jan. 7,2013, entitled, “A Method and System for Fusion of Fast Surprise andMotion-Based Saliency for Finding Objects of Interest in DynamicScenes”, which is a Continuation-in-Part application of U.S.Non-Provisional application Ser. No. 12/982,713, filed on Dec. 30, 2010,entitled, “System for Identifying Regions of Interest in VisualImagery”, now issued as U.S. Pat. No. 8,774,517, which is aContinuation-in-Part application of U.S. Non-Provisional applicationSer. No. 12/214,259, filed on Jun. 16, 2008, entitled, “Visual Attentionand Segmentation System,” now issued as U.S. Pat. No. 8,363,939. U.S.Non-Provisional application Ser. No. 13/743,742 is ALSO aContinuation-in-Part application of U.S. Non-Provisional applicationSer. No. 13/669,269, filed on Nov. 5, 2012, entitled, “Motion-SeededObject Based Attention for Dynamic Visual Imagery”, which is aContinuation-in-Part application of U.S. Non-Provisional applicationSer. No. 12/214,259, filed on Jun. 16, 2008, entitled, “Visual Attentionand Segmentation System”, now issued as U.S. Pat. No. 8,363,939, whichis a Continuation-in-Part application of U.S. Non-Provisionalapplication Ser. No. 11/973,161, filed on Oct. 4, 2007, entitled,“Visual Attention and Object Recognition System”, now issued as U.S.Pat. No. 8,165,407. U.S. Non-Provisional application Ser. No. 13/743,742is ALSO a Non-Provisional patent application of U.S. ProvisionalApplication No. 61/589,761, filed in the U.S. on Jan. 23, 2012, titled,“A Method and System for Fusion of Fast Surprise and Motion-BasedSaliency for Finding Objects of Interest in Dynamic Scenes.”

This is ALSO a Continuation-in-Part application of U.S. Non-Provisionalapplication Ser. No. 14/203,256, filed in the United States on Mar. 10,2014, entitled, “Graphical Display and User-Interface for High-SpeedTriage of Potential Items of Interest in Imagery”, which is aNon-Provisional application of U.S. Provisional Application No.61/779,320, filed in the United States on Mar. 13, 2013, entitled,“Graphical Display and User-Interface for High-Speed Triage of PotentialItems of Interest in Imagery.”

This is ALSO a Continuation-in-Part application of U.S. Non-Provisionalapplication Ser. No. 13/669,269, filed in the United States on Nov. 5,2012, entitled, “Motion-Seeded Object Based Attention for Dynamic VisualImagery”, which is a Continuation-in-Part application of U.S.Non-Provisional application Ser. No. 12/214,259, filed in the UnitedStates on Jun. 16, 2008, entitled, “Visual Attention and SegmentationSystem”, now issued as U.S. Pat. No. 8,363,939.

GOVERNMENT LICENSE RIGHTS

This invention was made with government support under U.S. GovernmentContract Number W31P4Q-08-C-0264. The government has certain rights inthe invention.

BACKGROUND OF THE INVENTION

(1) Field of Invention

The present invention relates to a system for surveillance and, moreparticularly, to a system for surveillance by integrating radar with apanoramic staring sensor.

(2) Description of Related Art

There are several existing surveillance systems that combine camera andradar which are designed for ground-level surveillance. Among them are(1) the Night Vision Labs Cerberus Scout manufactured by MTEQ located at140 Technology Park Drive, Kilmarnock, Va.; (2) the Blighter Explorermanufactured by Blighter Surveillance Systems located at The PlextekBuilding, London Road, Great Chesterford, Essex, CB10 1NY, UnitedKingdom; (3) the Honeywell Radar Video Surveillance (RVS) systemmanufactured by Honeywell, which is located at 2700 Blankenbaker Pkwy,Suite 150, Louisville, Ky. 40299; and (4) the U.S. Army's COSFPS (a.k.a.the “Kraken”). While designed to be deployed in different situations,these systems all share the common base configuration of a smallground-scanning radar that scans for targets, a camera or cameras (e.g.,electro-optical (EO) and infrared (IR)) that can mechanically slew andzoom to regions of interest (most likely as the result of a radarmessage), and an operator console that allows a human operator to eitherautomatically slew to radar hits or examine other locations by manuallycontrolling the camera.

Any system with a radar and single movable camera configuration willshare the same limitations. The cameras in these systems are reactive,meaning that the operator slews the camera to the location of a possibletarget only after the system registers a radar hit. Because of this,there is an unavoidable delay between the triggering of an event inradar and the operator's examination of the event. At minimum, thisdelay is the time required to slew the camera to the new location, whichmight be enough time for the target to change positions, though askilled operator might be able to “search” the regions surrounding thelocation of the radar hit and pick up the target. However, a moredangerous scenario is one in which multiple radar messages are receivedin quick succession of one another. In this case, the system operatormust choose which targets to attend to first, quickly analyze theregion, and then repeat with the subsequent radar hit locations. It iseasy to imagine a scenario in which the operator is swamped bysimultaneous radar hits and cannot examine them all in a timely manner.In this situation, targets will escape the vicinity of their radarlocation.

Thus, a continuing need exists for an advanced panoramic staring sensorthat covers a wide field-of-view and continuously records the entirepanorama in order to ensure that a radar target is never missed.

SUMMARY OF THE INVENTION

The present invention relates to a system for surveillance and, moreparticularly, to a system for surveillance by integrating radar with apanoramic staring sensor. The system comprises one or more processorsand a memory having instructions such that when the instructions areexecuted, the one or more processors perform multiple operations. Thesystem captures a set of image frames of a field-of-view of a sceneusing a multi-camera panoramic staring sensor. The field-of-view of thescene is scanned with a radar sensor to detect an object of interest. Aradar detection is received when the radar sensor detects the object ofinterest. Based on the radar detection, a radar message indicating thepresence of the object of interest is generated. Each image frame in theset of image frames is marked with a timestamp. The set of image framesis stored in a frame storage database. The set of radar-basedcoordinates from the radar message is converted into a set ofmulti-camera panoramic sensor coordinates. A video clip comprising asequence of image frames in the set of image frames corresponding intime to the radar message is created. The video clip is displayed,wherein the video clip displays the object of interest.

In another aspect, each image frame in the set of image frames iscompared to a background model with an active cognitive processormodule. The active cognitive processor module detects at least onecognitive detection in an image frame, wherein the at least onecognitive detection corresponds to a region of the scene that deviatesfrom the background model and represents the object of interest. Theactive cognitive processor module assigns a cognitive score and abounding box to each cognitive detection to aid in user analysis. Ahigher cognitive score corresponds to a greater deviation from thebackground model, and the bounding box surrounds the object of interest.The cognitive detections having the highest cognitive scores are storedin the frame storage module.

In another aspect, the controller module manages cognitive detectionsaccording to the following. A list of cognitive detections having thehighest cognitive scores is requested from the active cognitiveprocessor module. For each cognitive detection in the list, a sequenceof image frames comprising the image frame corresponding to thecognitive detection and a plurality of image frames before and after theimage frame corresponding to the cognitive detection is requested fromthe capture and recording module. For each cognitive detection in thelist, a video sequence corresponding to the time of the cognitivedetection is constructed from the sequence of image frames. For eachcognitive detection in the list, the video sequence is sent to the userinterface for user analysis.

In another aspect, a cognitive score for a region of the field-of-viewof the scene in which the radar message originated is retrieved from theactive cognitive processor module.

In another aspect, objects of interest are detected, in parallel, withboth the active cognitive processor module and the radar sensorindependently.

In another aspect, the bounding box is used to perform a classificationof the cognitive detection in a classification module using objectrecognition. A tracker is applied to the bounding box, and the boundingbox is tracked across image frames using a tracking module. The trackeris used to switch between a radar message location of the object ofinterest to a current position of the object of interest in thefield-of-view of the scene.

In another aspect, the controller module forwards the video clip to areactive cognitive processor module, wherein the reactive cognitiveprocessor module performs the following operations. The image frames inthe video clip are compared to a background model. At least onecognitive detection is detected in at least one image frame in the videoclip, wherein the cognitive detection corresponds to a region of thescene that deviates from the background model and represents the objectof interest. A cognitive score and a bounding box are assigned to eachcognitive detection to aid in user analysis, wherein a higher cognitivescore corresponds to a greater deviation from the background model, andthe bounding box surrounds the object of interest.

In another aspect, a plurality of multi-camera panoramic staring sensorsis used to continuously capture the set of image frames of thefield-of-view of the scene. A plurality of radar sensors are used todetect the object of interest to enable the system to scale up thefield-of-view to any predetermined value up to a 360-degreefield-of-view.

In another aspect, the present invention also comprises a method forcausing a processor to perform the operations described herein.

Finally, in yet another aspect, the present invention also comprises acomputer program product comprising computer-readable instructionsstored on a non-transitory computer-readable medium that are executableby a computer having a processor for causing the processor to performthe operations described herein.

BRIEF DESCRIPTION OF THE DRAWINGS

The objects, features and advantages of the present invention will beapparent from the following detailed descriptions of the various aspectsof the invention in conjunction with reference to the followingdrawings, where:

FIG. 1 is a flow diagram of a system for surveillance that integratesradar with a panoramic staring sensor according to the principles of thepresent invention;

FIG. 2A is an isometric-view illustration of a panoramic staring sensoraccording to the principles of the present invention;

FIG. 2B is a top-view illustration of a panoramic staring sensoraccording to the principles of the present invention;

FIG. 3 is an illustration of a panoramic staring sensor field-of-viewaccording to the principles of the present invention;

FIG. 4 illustrates scalability of the system according to the principlesof the present invention;

FIG. 5 is a table depicting detection results of the present inventionaccording to the principles of the present invention;

FIG. 6 is an illustration of a data processing system according to theprinciples of the present invention; and

FIG. 7 is an illustration of a computer program product according to theprinciples of the present invention.

DETAILED DESCRIPTION

The present invention relates to a system for surveillance and, moreparticularly, to a system for surveillance by integrating radar with apanoramic staring sensor. The following description is presented toenable one of ordinary skill in the art to make and use the inventionand to incorporate it in the context of particular applications. Variousmodifications, as well as a variety of uses, in different applicationswill be readily apparent to those skilled in the art, and the generalprinciples defined herein may be applied to a wide range of embodiments.Thus, the present invention is not intended to be limited to theembodiments presented, but is to be accorded with the widest scopeconsistent with the principles and novel features disclosed herein.

In the following detailed description, numerous specific details are setforth in order to provide a more thorough understanding of the presentinvention. However, it will be apparent to one skilled in the art thatthe present invention may be practiced without necessarily being limitedto these specific details. In other instances, well-known structures anddevices are shown in block diagram form, rather than in detail, in orderto avoid obscuring the present invention.

The reader's attention is directed to all papers and documents which arefiled concurrently with this specification and which are open to publicinspection with this specification, and the contents of all such papersand documents are incorporated herein by reference. All the featuresdisclosed in this specification, (including any accompanying claims,abstract, and drawings) may be replaced by alternative features servingthe same, equivalent or similar purpose, unless expressly statedotherwise. Thus, unless expressly stated otherwise, each featuredisclosed is one example only of a generic series of equivalent orsimilar features.

Furthermore, any element in a claim that does not explicitly state“means for” performing a specified function, or “step for” performing aspecific function, is not to be interpreted as a “means” or “step”clause as specified in 35 U.S.C. Section 112, Paragraph 6. Inparticular, the use of “step of” or “act of” in the claims herein is notintended to invoke the provisions of 35 U.S.C. 112, Paragraph 6.

Please note, if used, the labels left, right, front, back, top, bottom,forward, reverse, clockwise and counter-clockwise have been used forconvenience purposes only and are not intended to imply any particularfixed direction. Instead, they are used to reflect relative locationsand/or directions between various portions of an object. As such, as thepresent invention is changed, the above labels may change theirorientation.

Before describing the invention in detail, first a list of citedliterature references used in the description is provided. Next, adescription of various principal aspects of the present invention isprovided. Subsequently, an introduction provides the reader with ageneral understanding of the present invention. Finally, specificdetails of the present invention are provided to give an understandingof the specific aspects.

(1) List of Incorporated Cited Literature References

The following references are cited throughout this application. Forclarity and convenience, the references are listed herein as a centralresource for the reader. The following references are herebyincorporated by reference as though fully included herein. Thereferences are cited in the application by referring to thecorresponding literature reference number, as follows:

-   1. van den Broek, B.; Burghouts, G.; van den Broek, S.; Smith, A.;    Hagen, R.; Anitori, L.; and van Rossum, W., “Automatic Detection of    Hostile Behavior”, Proc SPIE vol 7480 74800R-1, 2009.-   2. Schwering, P. B. W.; Lensen, H. A.; van den Broek, S. P.; den    Hollander, R. J. M.; van der Mark, W.; Bouma, H.; and Kemp, R. A.    W., “Application of Heterogeneous Multiple Camera System with    Panoramic Capabilities in a Harbor Environment”, Proc SPIE vol 7481    74810C-1, 2009.-   3. Kyungnam Kim, Thanarat H. Chalidabhongse, David Harwood, and    Larry Davis, “Realtime foreground-background segmentation using    codebook model,” Real-Time Imaging, Volume 11, Issue 3, June 2005.-   4. Feng, G.; Tian, W.; Huang, C.; Liu, T.; and Zhang, S., “Wide    Field of View CCD Camera Based on Multi-Sensors Image Mosaics”    Congress on Image and Signal Processing (CISP) 2: 432-435, 2008.-   5. Gerson, A. D.; Parra, L. C.; and Sajda, P. “Cortically Coupled    Computer Vision for Rapid Image Search”. IEEE Transactions on Neural    Systems and Rehabilitation Engineering, 14(2): 174-179, 2006.-   6. Huber, David J. and Khosla, Deepak. “Bio-Inspired Surprise for    Real-Time Change Detection in Visual Imagery.” SPIE Defense,    Security, and Sensing, Orlando, Fla., 2011.-   7. Huber, David J.; and Khosla, Deepak. “A Bio-Inspired Method and    System for Visual Object-Based Attention and Segmentation.” SPIE    Defense, Security, and Sensing, Orlando Fla., 2010.-   8. Boris Babenko, Ming-Hsuan Yang, Serge Belongie “Robust Object    Tracking with Online Multiple Instance Learning” IEEE TPAMI, August    2011.-   9. Kim, Kyungnam; Chen, Yang; Honda, Alexander L.; Jeong, Changsoo    S.; Cheng, Shinko Y.; Zhang, Lei; Khosla, Deepak; and Kubena,    Randall L. “Bio-Inspired Algorithms for Target Detection and    Classification in Airborne Videos.” AUVSI Unmanned Systems North    America. Las Vegas, Nev., USA. 2012.-   10. Khosla, Deepak; Huber, David J.; Bhattacharyya, Rajan; Daily,    Michael J. “Neurally-Inspired Rapid Detection of Sparse Objects in    Videos.” SPIE Defense, Security, and Sensing, Orlando Fla., 2010.

(2) Principal Aspects

The present invention has three “principal” aspects. The first is asystem for surveillance which integrates radar with a panoramic staringsensor. The system is typically in the form of a computer system,computer component, or computer network operating software or in theform of a “hard-coded” instruction set. This system may take a varietyof forms with a variety of hardware devices and/or sensors and mayinclude computer networks, handheld computing devices, cellularnetworks, satellite networks, and other communication devices. As can beappreciated by one skilled in the art, this system may be incorporatedinto a wide variety of devices that provide different functionalities.The second principal aspect is a method for surveillance whichintegrates radar with a panoramic staring sensor. The third principalaspect is a computer program product. The computer program productgenerally represents computer-readable instruction means (instructions)stored on a non-transitory computer-readable medium such as an opticalstorage device, e.g., a compact disc (CD) or digital versatile disc(DVD), or a magnetic storage device such as a floppy disk or magnetictape. Other, non-limiting examples of computer-readable media includehard disks, read-only memory (ROM), and flash-type memories.

The term “instructions” as used with respect to this invention generallyindicates a set of operations to be performed on a computer, and mayrepresent pieces of a whole program or individual, separable, softwaremodules. Non-limiting examples of “instructions” include computerprogram code (source or object code) and “hard-coded” electronics (i.e.,computer operations coded into a computer chip). The “instructions” maybe stored on any non-transitory computer-readable medium such as afloppy disk, a CD-ROM, a flash drive, and in the memory of a computer.

(3) Introduction

A wide field-of-view and rapid response to threats are criticalcomponents of any surveillance system. Field-of-view is normallyimplemented by articulating a camera: allowing it to swivel to pan andtilt, and actively zooming in on “interesting” locations. Since a singlecamera suffers from the “soda straw” problem, where only a small portionof the scene can be examined at any given time (leaving the rest of thescene unwatched), surveillance systems often employ a radar unit todirect the operator to likely targets. This provides direction to thesearch, but still poses a security risk, since potentially hazardousactivities might be occurring in an unwatched portion of the field ofview while the operator is investigating another incident (eithercoincidental or intentionally distracting). There are a number ofsecurity systems that combine camera and radar that are designed forground-level surveillance.

Any system with a radar and single movable camera configuration willshare the same limitations—the cameras in these systems are reactive,meaning that the operator slews the camera to the location of a possibletarget only after the system registers a radar hit. Because of this,there is an unavoidable delay between the triggering of an event inradar and the operator's examination of the event. At minimum, thisdelay is the time required to slew the camera to the new location, whichmight be enough time for the target to change positions, though askilled operator might be able to “search” the regions surrounding thelocation of the radar hit and pick up the target. However, for radartargets that arrive while the operator is examining another target, thisdelay might be increased beyond the operator's ability to locate thetarget. Additionally, pan-tilt-zoom units can be troublesome, sincemoving parts can break down.

By implementing a panoramic staring sensor (that constantly monitors theentire wide field-of-view at maximum resolution) and advanced softwaremodules to process, record, and present the surveillance footage, thesystem described herein can continuously monitor a wide field-of-viewand detect many more potential targets than the state-of-the-artradar-assisted single-camera system.

(4) Specific Details

A flow diagram depicting the system according to the principles of thepresent invention is shown in FIG. 1. An aspect of this system comprisesa multi-camera, panoramic visual sensor 100, a portable radar 102sensor, and various software modules for control and analysis. Inaddition, there are a number of aspects that employ optional modulesthat dramatically improve the ability of the system to detect targetsand maintain a low false alarm rate.

The flow of the system is as follows. The panoramic staring sensor 100is set up to cover an expanded field-of-view (e.g., 120 degrees), andthe radar 102 is configured to cover this same field-of-view. The radar102 scans the field-of-view, and the sensor 100 captures video frames(one from each camera per time step). Video capture is asynchronous withthe radar imaging. The frame data is passed into a capture and recordingmodule 104 that exists either in processing hardware or software, whereit is stamped with the time of capture and stored into buffer in memory(i.e., frame storage 106) for rapid retrieval.

Continuing with FIG. 1, in one aspect of the present invention, theframes are actively processed using a cognitive processor module 108. Inthe cognitive processor module 108, the frame is processed through acognitive algorithm to detect anomalies, changes, and motion.Non-limiting examples of cognitive algorithms that could be used by thesystem according to the principles of the invention include thosedescribed in Literature Reference Nos. 6 and 7, and U.S. applicationSer. No. 12/316,779, filed on Dec. 16, 2008, and entitled, CognitiveNeural Method for Image Analysis,” all of which are hereby incorporatedby reference as though fully set forth herein.

Each of these “detections” is assigned a score and a bounding box insoftware. Detections with the highest scores are stored in memory andare made available to other modules for subsequent processing. All ofthis happens independently from the rest of the system. At certain fixedintervals, the system controller (i.e., the controller module 110) canpoll the cognitive processor module 108 for its list of detections,collect video clips for these detections by accessing the frame storage106 (produced by the capture and recording module 104), and displaythese video clips to the user 112 (operator) through a user interface114 for analysis.

Radar messages are continuously monitored by a radar interpreter module116. For the purposes of this application, a “radar detection” refers towhen the radar has detected a target, and a “radar message” is a formatterm in which the radar detection has been packaged as a standardmessage format. When a radar message indicating the presence of a targethas been received, the radar interpreter module 116 converts theradar-based coordinates from the message into coordinates relevant tothe multi-camera, panoramic visual sensor 100. In one aspect of thepresent invention, an operator-defined region of interest (ROI) isemployed. In this aspect, radar messages falling outside apre-designated “listen area” may be ignored at this stage. After thisprocessing has occurred, the radar interpreter module 116 notifies thesystem controller module 110 that action must be taken. The controllermodule 110 will then use the time data on the radar message to learn theproper frame number in the video, and then extract a video clipcorresponding to this radar target. This clip is then immediately shownto the user 112 for analysis (via, for example, a display device).

Additional alternative aspects of the present invention includeprocessing of video extracted for radar targets through a “reactive”cognitive processor module 118 to visually aid the user 112 in findingtargets within the clip (i.e., placing detection boxes) and compute aconfidence measure, or additional processing of cognitive and radartargets through algorithms intended to track or classify possibletargets in a tracking module 120 or a classification module 122,respectively. Non-limiting examples of tracking and classificationalgorithms are described in Literature Reference Nos. 8 and 9,respectively. Another aspect of the present invention involvesdisplaying cognitive and/or radar targets through a neural processormodule 124 using, for instance, the rapid serial visual presentation(RSVP) paradigm. RSVP measures the electroencephalograph (EEG) of theuser 112, allowing the system to reduce the false alarm rate byfiltering out imagery that fails to register the sufficient desiredneural response from the operator (e.g., the P300 waveform in EEG). RSVPis described in Literature Reference No. 10 and U.S. Pat. No. 8,285,052,entitled, “Image Ordering System Optimized Via User Feedback,” which ishereby incorporated by reference as though fully set forth herein.

(4.1) Panoramic Staring Sensor—Capture and Recording Module

A panoramic staring sensor 100 continuously monitors the expandedfield-of-view and records all frames as they are captured by the systemin the capturing and recording module 104. Although not limited thereto,the sensor 100 employed is similar to that discussed in Feng et al. (seeLiterature Reference No. 7).

As illustrated in FIGS. 2A and 2B, the sensor 100 is constructed from aseries of identical fixed focal length cameras 200 that have beenarranged to capture an extended field-of-view. FIG. 2A illustrates anisometric view of the sensor 100, while FIG. 2B illustrates a top viewof the sensor 100. For simplicity, the individual cameras 200 within thesensor 100 are referred to as “cameras” and the entire panoramic staringsensor containing multiple cameras is referred to as the “sensor”.

FIG. 3 depicts a panoramic staring sensor field-of-view. In thisinstance, each of the six cameras 200 in the sensor 100 captures aportion 300 (numbered 1, 2, 3, 4, 5, and 6) of the entire field-of-view302 with a marginal amount of overlap between cameras 200. The result isa panoramic capture of the entire field-of-view 302 at each time step,where each individual camera 200 covers a fixed portion 300 of thefield-of-view 302. The individual cameras 200 exhibit a limited amountof overlap in their respective coverages to ensure that all blind spotsare removed. Since the cameras 200 are incapable of zooming, each camera200 has sufficiently high resolution that it can detect targets 304 at adesired distance (which should be at least the functional range of theradar so that radar targets can be verified). The sensor 100 is referredto as a “panoramic staring sensor” because it consists of smallercameras 200 arranged to capture a panorama, and because it continuouslyrecords the entire field-of-view 302 at all times.

A capture and recording module (depicted as element 104 in FIG. 1) isassigned to each individual camera 200 in the panoramic staring sensor100 array. Each of these modules operates completely independently ofthe others and captures frames from the designated sensor (i.e., camera200), recording them in a circular buffer of predetermined length (e.g.,100 frames). The frames are marked with a timestamp and indexed in a waythat allows for random access and retrieval by the other modules and forcorrelating with the reports from radar during image retrieval. Toconserve communications bandwidth, the frames are stored uncompressed intheir native format (i.e., Bayer formatted images are stored in Bayerformat). The images are converted out of Bayer format in the othermodules after they have been transmitted.

(4.4) Portable Radar and Radar Interpreter Module

Embodiments according to the principles of the present invention mayemploy a small Doppler radar (depicted as element 102 in FIG. 1) unitwith a fixed scan sector that covers the entire field-of-view of thepanoramic staring sensor (depicted as element 100 in FIG. 1). The radar(FIG. 1, 102) is assumed to contain software and/or hardware modulesthat are able to convert the physics of the radar signals and delaysinto a practical latitude-longitude-elevation format for use by theradar interpreter module (depicted as element 116 in FIG. 1). Theobjective of the radar interpreter module (FIG. 1, element 116) is toconvert the coordinate information received from radar messages into thecamera coordinate system, which allows extraction of video from thecapture and recording module (depicted as element 104 in FIG. 1) foruser/operator analysis. This is carried out through a complex processthat involves calibration of the camera sensor (FIG. 1, 100) and radar(FIG. 1, 102) with regards to each other.

The radar system sends position reports of targets detected by theequipment. These reports include a position specified as latitude,longitude, and elevation above mean sea level, which is generated from adistance and heading measurement from a Doppler radar system. The radarsystem then computes the latitude and longitude of the target based onthe position of the radar (FIG. 1, element 102) unit and itsorientation. Elevation is determined, for example, by using digitalterrain elevation map data (DTED). Accuracy of the position reports isaffected by how carefully the radar operator sets the position andorientation of the radar unit in its software. The radar interpretermodule 116 listens to the messages sent by other tactical systems tomaintain a list of other active sensors and note any reported targets.

There are two main components to calibrating the relation between thesensor 100 and the real world. These are camera intrinsics and a mappingfrom the external world. There are two parts to calibration of theintrinsic camera parameters, consisting of the individual cameracharacteristics and the relative alignment between cameras. Theindividual camera characteristics are defined by the focal length of thelens, the spacing of pixels in the imaging hardware, and the pixeldimensions of the array. These features determine the angular positionof each pixel in the image with respect to the center of the image inone camera. The alignment between cameras could either be adjusted to aspecified alignment during manufacturing or measured after manufacturingby locating features that appear in the overlapping area of the coverageof adjacent cameras. In either case, the result is a table of theoverlap and vertical offset between adjacent pairs of cameras that isused to generate a sensor-wide mapping between pixels and angularpositions.

In the most general case, mapping the external world into cameracoordinates requires establishing the roll, pitch, and yaw of the sensor100. Because significant roll and pitch, especially in a wide fieldimaging system, result in tilted images, curved horizon lines, and messytrigonometric corrections, the sensor 100 is normally set up to be aslevel as possible. To accommodate corrections for residual amounts ofpitch and roll and to determine yaw, world calibration is based on thelatitude, longitude and elevation of three points. These points are forthe imaging system itself and two landmarks within the field-of-view. Inthe case of the landmarks, the pixel position of the landmark in thecamera array (i.e., sensor; FIG. 1, element 100) is also required. Usingthis information, a table giving the heading angle and elevation angleof the center of the field-of-view for each camera is created. Forcameras containing landmarks, this is computed using the inverse of thecalculation given below to determine the pixel position of a target. Forother cameras, the central position is based on linear interpolation orextrapolation from the known centers with minor adjustments for theoverlap and vertical misalignment measured during sensor intrinsiccalibration.

To calculate the pixel position of a given target, standard formulas forgreat circles on a sphere are used to compute the heading measurement,H, and distance, dist, to target from latitude/longitude of imager andtarget. Difference in altitude and distance between imager and target isused to compute elevation angle to target, E, where el_(t) and el_(c)represent the elevation of the target and camera, respectively, and atan denotes the arctangent function according to the following:

$E = {{{a\tan}( \frac{{el}_{t} - {el}_{c}}{dist} )}.}$

Given the calculated heading and elevation angles, the system calculatesa pixel position (pix_(x), pix_(y)) for the target in each camera.Typically, only one camera will result in a pixel position within itsarray, although some targets will fall in an overlap between adjacentcameras and depending on radar coverage, other targets may fall outsideall sensors. In the current system, when a target falls within thecamera overlap region, the system returns the coordinate belonging tothe camera on the right of the overlapping pair.

For each camera, i, the camera coordinate of a given target can becomputed. In this case, CX and CY are tables of the heading andelevation of the center of view, respectively, and each camera is Wpixels wide and Y pixels high, FL is the lens focal length inmillimeters (mm), and P is the pixel pitch in mm. The camera coordinateis computed according to the following:

${pix}_{x} = {\frac{W}{2} + {( \frac{FL}{P} ){\tan( {H - {CX}_{i}} )}}}$

${pix}_{y} = {\frac{V}{2} + {( \frac{FL}{P} ){{\tan( {E - {CY}_{i}} )}.}}}$These pixel positions may be adjusted slightly based on an estimate ofsensor roll. However, in practice, other errors in the overall system,such as radar position, heading, radar range, and heading measurements,are likely to be as large. If the pixel position computed falls outsidethe camera sensor array, the target is not visible in that camera.

(4.5) Cognitive Processor

Referring to FIG. 1, the cognitive processor module 108 providesanalysis of the video from the scene captured by the capturing andrecording module 104 and detects anomalies in the video. These“cognitive detections” are regions of the scene that deviate from abackground model 126 that the cognitive processor module 108 learns byanalyzing the scene for a fixed number of frames. Each cognitivedetection is described as a bounding box with a score that describes themagnitude of the deviation from a background model 126. The cognitiveprocessor module 108 allows easy ranking of potential items of interestand can aid the user 112 in identification of threats. This module isnot required, but generally improves the overall detection performanceof the system.

In alternative aspects of the system, the system may have an “active”cognitive processor module (represented by element 108 in FIG. 1) or a“reactive” cognitive processor module (represented by element 118 inFIG. 1). Systems with both an “active” cognitive processor module 108and a “reactive” cognitive processor module 118 are also possible. Foreither “active” or “reactive” types, the cognitive processor module (108or 118) continuously analyzes the image data captured by the panoramicstaring sensor 100 and executes anomaly and change detection algorithmson the data, looking for motion or other “salient” features that do notagree with that established in some background model 126. Descriptionsof anomaly and change detection algorithms can be found in U.S.application Ser. No. 14/203,256, entitled, “Graphical Display andUser-Interface for High-Speed Triage of Potential Items of interest inImagery”; U.S. application Ser. No. 12/982,713, entitled, “System forIdentifying Regions of Interest in Visual Imagery”; and U.S. applicationSer. No. 13/669,269, entitled, “Motion-Seeded Object Based Attention forDynamic Visual Imagery,” which are hereby incorporated by reference asthough fully set forth herein.

At the start of a run, the “active” cognitive processor module 108 willmonitor the scene for a fixed number of frames (e.g., 50 frames) andbuild a statistical model (i.e., background model 126) of the expectedcontent of the scene. After this background model 126 is created, it iscompared to each new incoming frame, which yields a set of “foreground”pixels that violate this background model 126 and represent anunexpected occurrence. These pixels are filtered and grouped by blobextraction algorithms, such as region-based image segmentation andconnected-component labeling, and returned as a rectangular region thatindicates the center, width, and height of the detection with a scorethat corresponds to how far from the background model 126 the detectionhas deviated.

In one aspect of the system, the panoramic staring sensor 100 may havean attached “active” cognitive processor module 108, which processes allframes and determines a set of “cognitive detections” independent of allother systems. In this aspect, the background model 126 must only becomputed once at the start of processing or whenever the sensor 100 ismoved. In the “active” configuration, the cognitive processor module 108computes on all incoming frames and records the highest scores for eachregion in the image (e.g., quadrants) over a given time interval (e.g.,30 seconds). The cognitive detections are sent to the controller module110 at these fixed intervals. The cognitive detections are treated likeradar detections, with video extraction and reporting to the user 112occurring at routine intervals. All cognitive detections are imprintedwith a score and detection rectangle that can be displayed in the userinterface 114 on top of the video clip, at the user's 112 discretion.

In another aspect of the system, the cognitive processor module may workin a “reactive” configuration (depicted as element 118 in FIG. 1). Inthis configuration, the cognitive processor module 118 is used tocompute scores and target rectangles for radar detections, but does notindependently produce new targets for the user 112 to analyze. The“reactive” cognitive processor module (FIG. 1, element 118) can beimplemented in one of two ways: prior to the receipt of the radar targetor as a response to the receipt of the radar message.

To implement the “reactive” cognitive processor module 118 prior toreceipt of a radar message, each frame must be processed by thecognitive algorithm as with the active system (element 108). Scores arecontinuously computed for each region of the image. When a radar targetis received, the system finds the score of the region that contains thecamera coordinates of the radar target. This method has the benefit ofnot requiring extra processing after the radar target message has beenreceived; however, this method cannot provide a bounding rectangle forthe target.

To implement the “reactive” cognitive processor module 118 after thereceipt of a radar message, frames are extracted for the time that theradar message was received and for several frames prior (e.g., thirtyframes). These frames are passed through the “reactive” cognitiveprocessor module 118. The detection algorithm trains on the first fewframes and then processes the remaining frames for change. While thismethod requires nominal additional processing time, it has the benefitof providing the user 112 with both a score and a target boundingrectangle for each radar target. This provides the user 112 with extrainformation that he or she can use to determine the validity of a giventarget.

In a third aspect of the system, the system may contain both an “active”and “reactive” cognitive processor modules (represented by elements 108and 118 in FIG. 1, respectively). In this case, the “active” cognitiveprocessor module 108 functions exactly as it does in the firstaforementioned aspect: processing all frames as they are captured andlogging the strongest detections over a period of time for thecontroller module 110 to request. Likewise, the “reactive” cognitiveprocessor module 118 works on frames that correspond to radar messages.The key difference in this aspect is that the “reactive” cognitiveprocessor module 118 may be able to use the background model 126computed by the “active” cognitive processor module 108, rather thancomputing a brand new background model for each radar detection. Sincethe background model 126 is built on a per-pixel basis, it is possiblefor the controller module 110 to request the background model 126 foreach pixel from the “active” cognitive processor 108. This permits the“reactive” cognitive processor module 118 to run more efficiently andreturn more frames with detection boxes, since frames that contribute tobackground model training cannot yield detection rectangles.

(4.6) Controller Module

The controller module 110 drives the timing of the system and mediatescommunication between all of the other modules. In the baseline aspectof the system, the controller module 110 acts as a simple intermediarybetween the radar interpreter module 116, the frame storage 106, and theuser interface 114. As each radar message is received, the controllermodule 110 places it into a queue that is continuously polled forentries that do not contain image data. Any type of scheduler may beemployed for this process, but it is often adequate to employ a FIFO(first in, first out) queue in which the oldest radar target that doesnot yet have image data will be processed before newer radar targets.For each radar message in the queue, the controller module 110 performsa lookup on the proper frame number within the frame storage 106database corresponding to the time of the radar message, and thenrequests a series of frames at and around the time of the radar message.

The frame window can be adjusted based on the type of surroundings andexpected targets. For example, the controller module 110 may requestthirty frames, where twenty of those frames may be from before the radarhit to give the operator the opportunity to see the events that led upto the radar message. The remaining ten frames can be obtained fromafter the radar message, centering the message somewhere in the middle.However, one must remember that obtaining frames after the radar messagerequires a wait before the user 112 can examine the video. After thevideo frames are obtained from the capture and recording module 104, thevideo clip is sent to the user interface 114 for user 112 analysis. Inaspects of the system that include an “active” cognitive processormodule 108 but not a “reactive” cognitive processor module 118, thecontroller module 110 will retrieve a cognitive score for the region inwhich the radar message originated. In this case, radar detection videoclips are displayed to the user 112 immediately upon receipt of imageryand cognitive score.

In aspects of the system that employ an “active” cognitive processormodule 108, the role of the controller module 110 is expanded to managecognitive detections as well as radar detections. In this instance, thecontroller module 110 will monitor the frame count and request the topdetections from the “active” cognitive processor module 108. For eachcognitive detection in this list, the controller module 110 will requestimage information in the immediate vicinity of the cognitive detectionat the appropriate frame number and construct a short video sequence ofthe scene at the time of the detection. The detection rectangle may besuperimposed on this video for analysis by the user 112. Once the videoclips for all cognitive detections have been fetched from the captureand recording module 104, the controller module 110 sends them to theuser interface 114 for user 112 analysis.

In aspects of this system that employ a “reactive” cognitive processormodule 118, the controller module 110 must forward the video clip foreach radar detection to the “reactive” cognitive processor module 118.The controller module 110 sends the results from cognitive processing tothe user interface 114 for user 112 analysis. In this aspect, radardetection video clips are displayed to the user 112 immediately uponcompletion of the cognitive processing.

The controller module 110 can also support additional modules to assistthe user 112 in the task of analyzing detections. After the controllermodule 110 has requested video frames for either a cognitive or radardetection, further processing, such as the application of additionalcognitive algorithms, tracking with the tracking module 120, orclassification with the classification module 122, may be applied. Theseoptions are described below.

(4.7) Additional Processing

Alternative aspects of the present invention may be implemented throughthe addition of optional processing modules. The objective of thisprocessing is to enhance the user's 112 experience with the system, aidin finding threats, and eliminate false alarms from radar and cognitivedetections. Below is a description of potential optional subsystems thatcan be added to the controller module 110 of the baseline aspect of thesystem. Many of these subsystems rely on the processing result of otheroptional modules and, therefore, might not be feasible in all possibleaspects of the system.

In aspects of the invention that include a cognitive processor (108and/or 110) that computes a bounding box for each cognitive and radardetection, one can apply additional processing in the form of trackingand classification/recognition of the object within the detection. Inthe tracking module 120, tracking software can be implemented as part ofthe controller module 110 to automatically track the most likely threatsfrom radar hits and the cognitive algorithm processing and present theseresults alongside the recorded results in the user interface 114. Theobjective of such a tracking system would be to allow the user 112 toswitch seamlessly between the recorded footage of the target from theradar detection and the target's current location. If the target isdeemed to be a threat, the current location is critical information.

An object recognition module can also be applied to the video datacorresponding to each cognitive or radar detection. In aspects of thesystem where a detection is marked with a bounding box (i.e., when thecognitive processor (elements 108 and/or 118) is employed), the contentsof this bounding box may be submitted to an objectrecognition/classification module 122 in order to apply a classificationlabel to the detection. These labels might be utilized by the user 112to rank or queue detections for analysis.

Another possible module that can be implemented in this system is atop-down biasing module that accepts user 112 input and modulates howthe system responds to radar hits within certain regions, or how thecognitive processor module (elements 108 and/or 118) works with framedata. For example, the user 112 could choose to completely ignorecognitive and radar detections from certain portions of the scene. Inthis instance, the cognitive processor (elements 108 and/or 110) wouldbe set to not process these regions of the scene, and all radar messagesthat originate from the ignored portion of the scene would not bepresented to the user 112. An inhibition operation could be implementedin the same way. However, instead of removing regions of the scene fromprocessing, the system would module the scores and queuing of cognitiveand radar detections in inhibited regions to move them to a lower spotin the order of processing and user analysis.

To reduce the false alarm rate of video clips sent to the user 112, oneaspect of the present invention could make use of a neural processormodule (depicted as element 124 in FIG. 1). The neural processor module(FIG. 1, element 124) employs a “human-in-the-loop” approach thatmeasures the operator's brain activity and uses this information tore-rank and filter out detections that are probable false alarms. Anexample of a method employed by the neural processor module (element124) is described in Literature Reference 8. For simplicity, the presentapplication only describes this process at a high level. Using thismethod, the video clips corresponding to cognitive and radar detectionsare displayed sequentially to the user 112 via a rapid serial visualpresentation (RSVP) paradigm, and the user's electroencephalograph (EEG)is recorded and processed by an algorithm, which looks for the presenceof the “P300” waveform in the EEG, which corresponds to a measure ofsurprise. The algorithm involves neural decoding that processesspatio-temporal EEG data to detect P300 and comes up with a score forpresence and confidence in P300 (see Literature Reference No. 10, U.S.Pat. No. 8,285,052, and U.S. Pat. No. 8,214,309, which are herebyincorporated by reference as though fully set forth herein, for detaileddescriptions of processing of the EEG signal). Each detection is given ascore based on the EEG at the time of the presentation of the video,which can be used for queuing or ranking of the detections in the userinterface (depicted as element 114 in FIG. 1). Cognitive or radardetections whose neural scores fall below some predetermined thresholdmay even be omitted from further user/operator analysis.

(4.8) User Interface

Once the radar has sent a message and video imagery has been procured bythe controller software, the radar hit can be displayed by the userinterface (depicted as element 114 in FIG. 1). In implementations of thesystem described herein, the “Threat Chip Display” (TCD) was employed toquickly display videos from cognitive and radar detections. LiteratureReference No. 4 provides a description of the TCD. Additionally, thisdisplay shows a panoramic image of the entire field-of-view and atop-down map with the latitude-longitude location of the targetsdisplayed. However, the user interface 114 is not necessarily limited tothe TCD implementation. A valid implementation of a user interface 114must be able to show cognitive and radar detections at full resolutionand accept feedback from the user 112 in the form of, for instance,target designation and algorithm biasing. The display should also allowthe user 112 to switch between the recorded footage and a “live” view ofthe region, as well as allow the user 112 to pan and zoom thefield-of-view to simulate the operation of single-camera systems.Additionally, the display should show a small, full-resolution video foreach target and its relative location in the entire field-of-view. Bothof these displays are essential to the user's ability to identify andrespond to possible threats in the field.

Radar detections can be added to the display as they are completed(i.e., asynchronously) or synchronously in bursts that occur at somefixed time interval (e.g., every thirty seconds). In aspects thatinclude an “active” cognitive processor module (FIG. 1, element 108),the cognitive detections are usually updated at fixed time intervals(i.e., no asynchronous operation).

As described above, the user (FIG. 1, element 112) can also adjust theregion of interest of the system using the user interface (FIG. 1,element 114). This is done by designating a region (or regions) in eachsensor frame as “out of bounds”. In order to be able to designate aregion (or regions) as out of bounds, the entire image is gridded intosmall regions (chips or rectangles), and the user can check on or offone or more regions via the user interface (FIG. 1, element 114). Thiscontrols whether they are in bound or out of bound (i.e., ignored forprocessing). This control is in addition to coarse region of interestselection via horizontal lines that span the full image.

A signal is then sent from the graphical user interface 114 to thecontroller module (FIG. 1, 110) that causes it to discard any radardetections that originate from these regions. This allows the system tofilter out radar detections from regions that are known to have a highfalse alarm rate or regions that the user 112 has predetermined will notcontain threats. The user 112 may also choose to mark detection videosas “inhibit” or “ignore”. Doing so affects the cognitive and radarprocessing of detections within the controller module 110.

(4.9) Scalability of the System

A key feature of the present invention is its ability to scale up fromits default field-of-view to any arbitrary value, up to a full360-degree field-of-view. As illustrated in FIG. 4, an alternate aspectof the invention involves implementing additional panoramic staringsensors 100 (and optionally, additional radars to decrease the scantime) to cover the expanded 360-degree field-of-view. As a non-limitingexample, a 360-degree system can be made from the present invention byaligning three, 120-degree systems to cover the entire field-of-view.This non-limiting configuration employs a single radar 102 that scans inone direction and rotates clockwise (as illustrated by the arrows),although a second radar could be used to scan in both directions atonce, reducing the scan time. Each 120-degree system has its own captureand recording module (CRM) 104 and “active” cognitive processor (CP)module 108 so that increasing the field-of-view does not affectprocessing speed or system response to targets. All systems can sharethe same controller module and user interface (not shown).

The implementation of additional panoramic staring sensors 100 does notaffect the processing speed or detection effectiveness of the system;the only difference between the baseline system and the expanded360-degree system shown in FIG. 4 is the number of detections presentedto the operator/user. This is because the processing hardware and memoryexpands with the field-of-view. Each panoramic staring sensor 100 isassigned a corresponding capture and recording module (CRM) 104 and an“active” cognitive processor module (CPM) 108, which ensures that theprocessing of individual frames increases linearly with field-of-viewwidth. The radar 102 must be configured to scan the entire 360-degreefield-of-view, and additional radars may be implemented to increase thescan frequency of any given region within the field-of-view.

In operation, expanding the field-of-view results in more cognitive andradar detections. The processing of these detections is carried outexactly the same as in the baseline system. The controller module(depicted as element 110 in FIG. 1) will request multiple frames ofvideo for detections resulting from radar detections (and cognitivedetections if the “active” cognitive processor module (FIG. 1, element108) is available). These videos will be passed through whateveradditional processing is available and presented to the user (FIG. 1,112) for inspection. Since the panoramic staring sensor (FIG. 1, element100) can capture the entire 360-degree field-of-view without slewing acamera, expanding the field-of-view does not add delay to thisprocedure. In aspects of the invention with the “active” cognitiveprocessor module 108, the controller module 110 will still request thetop cognitive detections and video frames at regular intervals andpresent them to the user 112. Due to the design of the user interface(FIG. 1, element 114) and lack of camera control, these additionaldetections do not increase the response time or the user's 112 abilityto analyze the detections in a timely manner.

The system described herein can also be scaled down to a smallerfield-of-view. This is carried in the baseline system by disablingsensors in the capture and recording module (FIG. 1, element 104) andthe controller module (FIG. 1, element 110) that correspond to regionsin the scene that are not important, which are not processed.Alternatively, panoramic staring sensors (FIG. 1, element 100) can bebuilt using fewer individual cameras in order to narrow thefield-of-view.

(4.10) Experimental Studies

Experimental studies were performed on the system described herein usinga combination of video recorded from the panoramic staring sensor andradar that was collected. A series of forty-five minute scenarios wereconducted in which dismounts and vehicles moved within a 120-degreefield-of-view in front of the camera/sensor. The vehicles and dismountsexhibited varying degrees of avoidance to being seen by thecamera/sensor. For the purposes of ground truth, each dismount andvehicle was equipped with a global positioning system (GPS) recorder,which continuously recorded the position of each dismount and vehicle. Afour-sensor 120-degree wide field-of-view camera was used to capturevideo of the scenario, and a portable radar unit was deployed to coverthe same 120-degree field-of-view. All scenarios contained a total of268 events.

As the scenario occurred, the video from each camera was processed witha change detection process that pulled out anomalies and scored eachregion in the video with a measure of how anomalous it was. The changedetection process is described in U.S. application Ser. No. 13/743,742,entitled, “A Method and System for Fusion of Fast Surprise andMotion-Based Saliency for Finding Objects of Interest in DynamicScenes,” which is hereby incorporated by reference as though fully setforth herein. “Anomalies” are defined as objects of interest in thevideos discovered by the change detection process.

Simultaneous to the change detection processing, the radar was run andtargets were retrieved by the system in real time. As each target wasdetected by the radar, the system converted the radar tilt-rangeinformation into latitude-longitude, and again into sensor and cameracoordinates (s; x, y) in the radar interpreter module. From the cameracoordinates, a short video clip (e.g., 5 frames) was obtained for a512-by-512 pixel region centered at the location of the detection (i.e.,the set of multi-camera panoramic sensor coordinates). Additionally, the“cognitive score” was copied from the cognitive processor module for thepurpose of ranking the radar targets in the user interface.

The targets were displayed to the user/operator in real-time as theywere detected by the radar. The method for display was a TCD layout. Atthis time, the user examined the video and determined which detectionswere from actual items of interest and which were not. At thirty-secondintervals, the user was also presented with the top fifty results fromthe cognitive algorithms over that time period.

A ground-truth for each scenario was constructed from the GPS data fromeach dismount and vehicle. The location of each dismount and vehicle wascomputed in sensor coordinates (s; x, y) for each frame fromlatitude-longitude coordinates using the same conversion method used toconvert the radar hits into camera coordinates. The GPS tracks andsequence frames may not be captured at exactly the same time; therefore,the GPS data was used for each target that provided the closest match intime of day to when the frame was captured. For instances where therewas a large difference in the camera coordinates for a given targetbetween GPS samples, linear interpolation between the two nearest GPSsamples was used.

The results of the experimental trials are displayed in the tabledepicted in FIG. 5. The table compares the present invention (Systemcolumn) with results of the state-of-the-art Cerberus Scout (Scoutcolumn) and CT2WS (CT2WS column) systems. Numbers listed are the pD(i.e., fraction of detections) for the given system for each day. Whilethe results for the present invention were computed from recorded videoand radar, the results from the Scout and CT2WS systems were obtainedduring the actual field test. While there is not any false alarm datafor these trials, one can see that the combination radar and panoramicstaring sensor system (with the “active” cognitive processor module)according to the principles of the present invention (System column) wasable to detect 95% of the available (i.e., all) targets, outperformingthe CT2WS system at 84%, and greatly outperforming the Cerberus Scoutsystem at 41%.

An example of a computer system 600 in accordance with one aspect isshown in FIG. 6. The computer system 600 is configured to performcalculations, processes, operations, and/or functions associated with aprogram or algorithm. In one aspect, certain processes and stepsdiscussed herein are realized as a series of instructions (e.g.,software program) that reside within computer readable memory units andare executed by one or more processors of the computer system 600. Whenexecuted, the instructions cause the computer system 600 to performspecific actions and exhibit specific behavior, such as describedherein.

The computer system 600 may include an address/data bus 602 that isconfigured to communicate information. Additionally, one or more dataprocessing units, such as a processor 604, are coupled with theaddress/data bus 602. The processor 604 is configured to processinformation and instructions. In one aspect, the processor 604 is amicroprocessor. Alternatively, the processor 604 may be a different typeof processor such as a parallel processor, or a field programmable gatearray.

The computer system 600 is configured to utilize one or more datastorage units. The computer system 600 may include a volatile memoryunit 606 (e.g., random access memory (“RAM”), static RAM, dynamic RAM,etc.) coupled with the address/data bus 602, wherein a volatile memoryunit 606 is configured to store information and instructions for theprocessor 604. The computer system 600 further may include anon-volatile memory unit 608 (e.g., read-only memory (“ROM”),programmable ROM (“PROM”), erasable programmable ROM (“EPROM”),electrically erasable programmable ROM “EEPROM”), flash memory, etc.)coupled with the address/data bus 602, wherein the non-volatile memoryunit 608 is configured to store static information and instructions forthe processor 604. Alternatively, the computer system 600 may executeinstructions retrieved from an online data storage unit such as in“Cloud” computing. In an embodiment, the computer system 600 also mayinclude one or more interfaces, such as an interface 610, coupled withthe address/data bus 602. The one or more interfaces are configured toenable the computer system 600 to interface with other electronicdevices and computer systems. The communication interfaces implementedby the one or more interfaces may include wireline (e.g., serial cables,modems, network adaptors, etc.) and/or wireless (e.g., wireless modems,wireless network adaptors, etc.) communication technology.

In one aspect, the computer system 600 may include an input device 612coupled with the address/data bus 602, wherein the input device 612 isconfigured to communicate information and command selections to theprocessor 600. In accordance with one aspect, the input device 612 is analphanumeric input device, such as a keyboard, that may includealphanumeric and/or function keys. Alternatively, the input device 612may be an input device(s) other than an alphanumeric input device, suchas the user interface, panoramic staring sensor and radar, or anycombination of devices that provide the functionalities as describedherein. In one aspect, the computer system 600 may include a cursorcontrol device 614 coupled with the address/data bus 602, wherein thecursor control device 614 is configured to communicate user inputinformation and/or command selections to the processor 600. In oneaspect, the cursor control device 614 is implemented using a device suchas a mouse, a track-ball, a track-pad, an optical tracking device, or atouch screen. The foregoing notwithstanding, in one aspect, the cursorcontrol device 614 is directed and/or activated via input from the inputdevice 612, such as in response to the use of special keys and keysequence commands associated with the input device 612. In analternative aspect, the cursor control device 614 is configured to bedirected or guided by voice commands.

In one aspect, the computer system 600 further may include one or moreoptional computer usable data storage devices, such as a storage device616, coupled with the address/data bus 602. The storage device 616 isconfigured to store information and/or computer executable instructions.In one aspect, the storage device 616 is a storage device such as amagnetic or optical disk drive (e.g., hard disk drive (“HDD”), floppydiskette, compact disk read only memory (“CD-ROM”), digital versatiledisk (“DVD”)). Pursuant to one aspect, a display device 618 is coupledwith the address/data bus 602, wherein the display device 618 isconfigured to display video and/or graphics. In one aspect, the displaydevice 618 may include a cathode ray tube (“CRT”), liquid crystaldisplay (“LCD”), field emission display (“FED”), plasma display, or anyother display device suitable for displaying video and/or graphic imagesand alphanumeric characters recognizable to a user.

The computer system 600 presented herein is an example computingenvironment in accordance with one aspect. However, the non-limitingexample of the computer system 600 is not strictly limited to being acomputer system. For example, one aspect provides that the computersystem 600 represents a type of data processing analysis that may beused in accordance with various aspects described herein. Moreover,other computing systems may also be implemented. Indeed, the spirit andscope of the present technology is not limited to any single dataprocessing environment. Thus, in one aspect, one or more operations ofvarious aspects of the present technology are controlled or implementedusing computer-executable instructions, such as program modules, beingexecuted by a computer. In one implementation, such program modulesinclude routines, programs, objects, components and/or data structuresthat are configured to perform particular tasks or implement particularabstract data types. In addition, one aspect provides that one or moreaspects of the present technology are implemented by utilizing one ormore distributed computing environments, such as where tasks areperformed by remote processing devices that are linked through acommunications network, or such as where various program modules arelocated in both local and remote computer-storage media includingmemory-storage devices.

An illustrative diagram of a computer program product embodying thepresent invention is depicted in FIG. 7. As a non-limiting example, thecomputer program product is depicted as either a floppy disk 700 or anoptical disk 702. However, as mentioned previously, the computer programproduct generally represents computer readable code (i.e., instructionmeans or instructions) stored on any compatible non-transitory computerreadable medium.

What is claimed is:
 1. A system for surveillance, the system comprising:one or more processors and a non-transitory computer-readable mediumhaving executable instructions encoded thereon such that when executed,the one or more processors perform operations of: storing a set of imageframes of a field-of-view of a scene captured using a multi-camerapanoramic staring sensor in a frame storage database, wherein each imageframe is marked with a time of image frame capture; generating a radardetection when a radar sensor detects an object of interest in thefield-of-view of the scene; based on the radar detection, generating aradar message, marked with a time of radar detection, indicating thepresence of the object of interest; for each radar detection, convertinga set of radar coordinates corresponding to the radar detection into aset of multi-camera panoramic sensor coordinates; creating a video clipcomprising a sequence of image frames in the set of image frames,wherein the times of image frame capture for the sequence of imageframes correspond to the times of radar detections; and displaying thevideo clip, wherein the video clip displays the object of interest. 2.The system as set forth in claim 1, wherein the one or more processorsfurther perform operations of: comparing, with an active cognitiveprocessor module, each image frame in the set of image frames to abackground model; detecting, with the active cognitive processor module,at least one cognitive detection in an image frame, wherein the at leastone cognitive detection corresponds to a region of the scene thatdeviates from the background model and represents the object ofinterest; assigning, with the active cognitive processor module, acognitive score and a bounding box to each cognitive detection to aid inuser analysis, wherein a higher cognitive score corresponds to a greaterdeviation from the background model, and the bounding box surrounds theobject of interest; and storing the cognitive detections having thehighest cognitive scores in the frame storage database.
 3. The system asset forth in claim 2, wherein the one or more processors further performoperations of managing cognitive detections according to the following:requesting a list of cognitive detections having the highest cognitivescores from the active cognitive processor module; for each cognitivedetection in the list, requesting a sequence of image frames comprisingthe image frame corresponding to the cognitive detection and a pluralityof image frames before and after the image frame corresponding to thecognitive detection from the capture and recording module; for eachcognitive detection in the list, constructing a video sequencecorresponding to the time of the cognitive detection from the sequenceof image frames; and for each cognitive detection in the list, sendingthe video sequence to the user interface for user analysis.
 4. Thesystem as set forth in claim 3, wherein the one or more processorsfurther perform an operation of retrieving from the active cognitiveprocessor module a cognitive score for a region of a field-of-view of ascene in which a radar detection originated.
 5. The system as set forthin claim 4, wherein the one or more processors further perform anoperation of detecting, in parallel, objects of interest with both theactive cognitive processor module and the radar sensor independently. 6.The system as set forth in claim 2, wherein the one or more processorsfurther performs operations of: using the bounding box to perform aclassification of the cognitive detection in a classification moduleusing object recognition; applying a tracker to the bounding box, andtracking the bounding box across image frames using a tracking module,wherein a user can utilize the tracker to switch between at least oneimage frame in the video clip corresponding to a radar detection to acurrent location of the object of interest.
 7. The system as set forthin claim 1, wherein the one or more processors further perform anoperation of forwarding the video clip to a reactive cognitive processormodule, wherein the reactive cognitive processor module performsoperations of: comparing the image frames in the video clip to abackground model; detecting at least one cognitive detection in at leastone image frame in the video clip, wherein the cognitive detectioncorresponds to a region of the scene that deviates from the backgroundmodel and represents the object of interest; and assigning a cognitivescore and a bounding box to each cognitive detection to aid in useranalysis, wherein a higher cognitive score corresponds to a greaterdeviation from the background model, and the bounding box surrounds theobject of interest.
 8. The system as set forth in claim 1, wherein theone or more processors further perform operations of: using a pluralityof multi-camera panoramic staring sensors to continuously capture theset of image frames of the field-of-view of the scene; and using aplurality of radar sensors to detect the object of interest to enablethe system to scale up the field-of-view to any predetermined value upto a 360-degree field-of-view.
 9. A computer-implemented method forsurveillance, comprising an act of: causing one or more processors toexecute instructions stored on a non-transitory memory such that uponexecution, the one or more processors performs operations of: storing aset of image frames of a field-of-view of a scene captured using amulti-camera panoramic staring sensor in a frame storage database,wherein each image frame is marked with a time of image frame capture;generating a radar detection when a radar sensor detects an object ofinterest in the field-of-view of the scene; based on the radardetection, generating a radar message, marked with a time of radardetection, indicating the presence of the object of interest; for eachradar detection, converting a set of radar coordinates corresponding tothe radar detection into a set of multi-camera panoramic sensorcoordinates; creating a video clip comprising a sequence of image framesin the set of image frames, wherein the times of image frame capture forthe sequence of image frames correspond to the times of radardetections; and displaying the video clip, wherein the video clipdisplays the object of interest.
 10. The method as set forth in claim 9,wherein the one or more processors further performs operations of:comparing, with an active cognitive processor module, each image framein the set of image frames to a background model; detecting, with theactive cognitive processor module, at least one cognitive detection inan image frame, wherein the at least one cognitive detection correspondsto a region of the scene that deviates from the background model andrepresents the object of interest; assigning, with the active cognitiveprocessor module, a cognitive score and a bounding box to each cognitivedetection to aid in user analysis, wherein a higher cognitive scorecorresponds to a greater deviation from the background model, and thebounding box surrounds the object of interest; and storing the cognitivedetections having the highest cognitive scores in the frame storagedatabase.
 11. The method as set forth in claim 10, wherein the one ormore processors further perform an operation of managing cognitivedetections according to the following: requesting a list of cognitivedetections having the highest cognitive scores from the active cognitiveprocessor module; for each cognitive detection in the list, requesting asequence of image frames comprising the image frame corresponding to thecognitive detection and a plurality of image frames before and after theimage frame corresponding to the cognitive detection from the captureand recording module; for each cognitive detection in the list,constructing a video sequence corresponding to the time of the cognitivedetection from the sequence of image frames; and for each cognitivedetection in the list, sending the video sequence to the user interfacefor user analysis.
 12. The method as set forth in claim 11, wherein thedata processor further performs an operation of retrieving from theactive cognitive processor module a cognitive score for a region of afield-of-view of a scene in which a radar detection originated.
 13. Themethod as set forth in claim 12, wherein the data processor furtherperforms an operation of detecting, in parallel, objects of interestwith both the active cognitive processor module and the radar sensorindependently.
 14. The method as set forth in claim 10, wherein the dataprocessor further performs operations of: using the bounding box toperform a classification of the cognitive detection in a classificationmodule using object recognition; applying a tracker to the bounding box,and tracking the bounding box across image frames using a trackingmodule, wherein a user can utilize the tracker to switch between atleast one image frame in the video clip corresponding to a radardetection to a current location of the object of interest.
 15. Themethod as set forth in claim 9, wherein the one or more processorsfurther perform an operation of forwarding the video clip to a reactivecognitive processor module, wherein the reactive cognitive processormodule performs operations of: comparing the image frames in the videoclip to a background model; detecting at least one cognitive detectionin at least one image frame in the video clip, wherein the cognitivedetection corresponds to a region of the scene that deviates from thebackground model and represents the object of interest; and assigning acognitive score and a bounding box to each cognitive detection to aid inuser analysis, wherein a higher cognitive score corresponds to a greaterdeviation from the background model, and the bounding box surrounds theobject of interest.
 16. The method as set forth in claim 9, wherein theone or more processors further performs operations of: using a pluralityof multi-camera panoramic staring sensors to continuously capture theset of image frames of the field-of-view of the scene; and using aplurality of radar sensors to detect the object of interest to enablethe system to scale up the field-of-view to any predetermined value upto a 360-degree field-of-view.
 17. A computer program product forsurveillance, the computer program product comprising computer-readableinstructions stored on a non-transitory computer-readable medium thatare executable by a computer having a processor for causing theprocessor to perform operations of: storing a set of image frames of afield-of-view of a scene captured using a multi-camera panoramic staringsensor in a frame storage database, wherein each image frame is markedwith a time of image frame capture; generating a radar detection when aradar sensor detects an object of interest in the field-of-view of thescene; based on the radar detection, generating a radar message, markedwith a time of radar detection, indicating the presence of the object ofinterest; for each radar detection, converting a set of radarcoordinates corresponding to the radar detection into a set ofmulti-camera panoramic sensor coordinates; creating a video clipcomprising a sequence of image frames in the set of image frames,wherein the times of image frame capture for the sequence of imageframes correspond to the times of radar detections; and displaying thevideo clip, wherein the video clip displays the object of interest. 18.The computer program product as set forth in claim 17, furthercomprising instructions for causing the processor to perform operationsof: comparing, with an active cognitive processor module, each imageframe in the set of image frames to a background model; detecting, withthe active cognitive processor module, at least one cognitive detectionin an image frame, wherein the at least one cognitive detectioncorresponds to a region of the scene that deviates from the backgroundmodel and represents the object of interest; assigning, with the activecognitive processor module, a cognitive score and a bounding box to eachcognitive detection to aid in user analysis, wherein a higher cognitivescore corresponds to a greater deviation from the background model, andthe bounding box surrounds the object of interest; and storing thecognitive detections having the highest cognitive scores in the framestorage database.
 19. The computer program product as set forth in claim18, wherein the one or more processors further perform an operation ofmanaging cognitive detections according to the following: requesting alist of cognitive detections having the highest cognitive scores fromthe active cognitive processor module; for each cognitive detection inthe list, requesting a sequence of image frames comprising the imageframe corresponding to the cognitive detection and a plurality of imageframes before and after the image frame corresponding to the cognitivedetection from the capture and recording module; for each cognitivedetection in the list, constructing a video sequence corresponding tothe time of the cognitive detection from the sequence of image frames;and for each cognitive detection in the list, sending the video sequenceto the user interface for user analysis.
 20. The computer programproduct as set forth in claim 19, further comprising instructions forcausing the processor to perform an operation of retrieving from theactive cognitive processor module a cognitive score for a region of afield-of-view of a scene in which a radar detection originated.
 21. Thecomputer program product as set forth in claim 20, further comprisinginstructions for causing the processor to perform an operation ofdetecting, in parallel, objects of interest with both the activecognitive processor module and the radar sensor independently.
 22. Thecomputer program product as set forth in claim 18, further comprisinginstructions for causing the processor to perform operations of: usingthe bounding box to perform a classification of the cognitive detectionin a classification module using object recognition; applying a trackerto the bounding box, and tracking the bounding box across image framesusing a tracking module, wherein a user can utilize the tracker toswitch between at least one image frame in the video clip correspondingto a radar detection to a current location of the object of interest.23. The computer program product as set forth in claim 17, wherein theone or more processors further perform an operation of forwarding thevideo clip to a reactive cognitive processor module, wherein thereactive cognitive processor module performs operations of: comparingthe image frames in the video clip to a background model; detecting atleast one cognitive detection in at least one image frame in the videoclip, wherein the cognitive detection corresponds to a region of thescene that deviates from the background model and represents the objectof interest; and assigning a cognitive score and a bounding box to eachcognitive detection to aid in user analysis, wherein a higher cognitivescore corresponds to a greater deviation from the background model, andthe bounding box surrounds the object of interest.
 24. The computerprogram product as set forth in claim 17, further comprisinginstructions for causing the processor to perform operations of: using aplurality of multi-camera panoramic staring sensors to continuouslycapture the set of image frames of the field-of-view of the scene; andusing a plurality of radar sensors to detect the object of interest toenable the system to scale up the field-of-view to any predeterminedvalue up to a 360-degree field-of-view.
 25. A system for surveillance,the system comprising: a multi-camera panoramic staring sensor; a radarsensor; one or more processors and a non-transitory computer-readablemedium having executable instructions encoded thereon such that whenexecuted, the one or more processors perform operations of: storing aset of image frames of a field-of-view of a scene captured using themulti-camera panoramic staring sensor in a frame storage database,wherein each image frame is marked with a time of image frame capture;generating a radar detection when the radar sensor detects a object ofinterest in the field-of-view of the scene; based on the radardetection, generating a radar message, marked with a time of radardetection, indicating the presence of the object of interest; for eachradar detection, converting a set of radar coordinates corresponding tothe radar detection into a set of multi-camera panoramic sensorcoordinates; creating a video clip comprising a sequence of image framesin the set of image frames, wherein the times of image frame capture forthe sequence of image frames correspond to the times of radardetections; and displaying the video clip, wherein the video clipdisplays the object of interest.